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Description of NAEP Writing Achievement Levels-Setting Process 
and Proposed Achievement Level Definitions 



Introduction 

American College Testing convened a meeting to set achievement levels on the 1992 
NAEP in writing July 24-28, 1992, in St. Louis, Missouri. The purpose of the meeting was 
to develop writing achievement levels for grades 4, 8, and 12 to be recommended for use 
in reporting the 1992 NAEP. 

Panelists 

Panelists were nominated and selected according to procedures and criteria described 
in the "Design Document for Setting Achievement Levels on the 1992 National 
Assessment of Educational Progress in Mathematics, Reading, and Writing," dated 
January 16, 1992. The relevant portion of that document is included as Appendix A. The 
nomination process resulted in a pool of 377 potential panelists, from which 66 panelists 
were selected and invited to participate. AH 66 persons, representing 27 states and the 
District of Columbia, attended the St. Louis meeting and participated in the achievement 
levels-setting process. 

States represented in the meeting included: 



Alabama (2) 
California (5) 
Colorado 
Connecticut 
Dist. of Columbia 
Kentucky (4) 
Florida 



Georgia (6) 
Indiana 
Iowa (3) 
Louisiana 
Maryland (2) 
Massachusetts (4) 
Michigan (2) 



Mississippi (2) 
Montana 
New Jersey (2) 
New Mexico (4) 
New York (3) 
North Carolina 
North Dakota 



Ohio (4) 

Pennsylvania (5) 
Tennessee 
Texas (3) 
Virginia 

West Virginia (2) 
Wisconsin (2) 



The panel was composed of 35 (53 percent) teachers, 9 (14 percent) non-teacher 
educators, and 22 (33 percent) noneducators, representing business and industry, 
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government service, parents, and the general public (see Table 1). Eighteen (27 percent) 
of the panelists were male and 16 (24 percent) were minority. Each panel member was 
nominated to serve on a particular grade-level panel based on background, and members 
were drawn from the grade-level pools of nominees to serve on a specific grade-level 
panel. Twenty-two panelists served at grade 4, 21 panelists served at grade 8, and 23 
panelists served at grade 12. 

Advance Materials 

Two of the elements most critical to the success of the process of setting achievement 
levels are that participants be thoroughly familiar with the methodology to be employed 
and that they have a sufficient understanding of key background materials. To that end, 
panelists were provided a set of materials 8 to 10 days before the meeting. These 
advance materials included: 

• a cover letter explaining their role in the meeting 

• a statement of the NAEP mission, principles, and objectives 

• a security agreement detailing procedures for handling and discussing secure test 
materials 

• a NAEP overview, "Consensus at Work" 

• a brief introduction to the modified Angoff process 

• a copy of the NAEP "Writing Framework" 

• a copy of NAGB's policy framework, "Setting Appropriate Achievement Levels for the 
National Assessment of Educational Progress" 

• a meeting agenda 
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Table 1 

Descriptive Data for Writing Nominees and Panelists 





iNUI 1 HI icco 




Total 


377 


66 


Grade 4 


32.6% 


33.3% 


Grade 8 


30.8 


31, 8 


Grade 12 


36.6 


34.8 


Panelist Type 






Teacher 


35.5% 


53.0% 


Non-Teacher Educator 


46.4 


13.6 


General Public 


18.0 


33.3 


Gender 






Male 


18.0% 


27.3 


Female 


82.0 


72.7 


Race/Ethnicitv 






White 


72.4% 


75.8% 


Black 


19.1 


18.2 


Asian 


2.9 


1.5 


Native American 


0.1 


0.0 


Hispanic 


4.5 


4.5 


No Data 


0.0 


0.0 


Reaion of Nominator 






West 


27.3% 


21.2% 


Central 


29.4 


19.7 


Southeast 


30.0 


31.8 


Northeast 


13.3 


27.3 


Community Tvpe of Nominator 






Los SES 


31.6% 


28.8% 


Not Low SES 


34.7 


57.6 


No Data 


33.7 


13.6 


District Size of Nominator 






50,000 or More 


30.0% 


30.3% 


Less Than 50,000 


36.3 


56.1 


No Data/Not Appropriate 


33.7 


13.6 
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The Achievement Levels-Setting Process 
O/ervie^ of NAGB/NAEP 

After being welcomed and reviewing the agenda (see Appendix B), panelists were 
given an overview of the NAEP program and of NAGB (see Figure 1). This presentation 
emphasized the consensus process used in determining the NAEP Writing Framework, 
the extensive item review process for NAEP Writing prompts, and the improvements made 
in the NAEP Writing Assessment over time. 

Overview of the Writing Framewort^ 

Following a brief presentation on the official definitions of the achievement levels 
(Basic, Proficient, Advanced) one of ACTs content experts presented an overview of the 
NAEP Writing Framework and discussed factors that influence writing^ and writing prompt 
difficulty, including type of writing required (i.e., narrative, persuasive, informative). The 
purpose of the presentation was to focus panelists' attention on the Writing Framework 
and the NAEP Writing Assessment and to emphasize the fact that panelists' work was 
directly related to the NAEP Writing Assessment, not to the whole domain of writing. 

Completion of NAEP Writing Assessment 

All panelists completed an appropriate grade-level form of the NAEP Writing 
Assessment and reviewed their completed examinations using scoring guides. The 



^ Information presented in the overview was developed in consultation with 
representatives from the National Council of Teachers of English and others 
experienced with the NAEP Writing Frameworks during a meeting held in Boulder, 
Colorado June 3, 1992. 
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purpose of this exercise was to familiarize panelists with the test content and scoring 
protocols. 

Operationalizing Definitions and Identification of Sample Papers 

Working in small groups of 5 to 6, panelists expanded and operationalized the NAGB 
policy definitions of Basic, Proficient, and Advanced (Appendix C). The small groups (four 
for each grade--4, 8, and 12) were allowed to "brainstorm" about what student 
performance at each achievement level should be, using the Writing Framework, the 
policy definitions, and their experience in completing the NAEP Writing Assessment as 
guides. One person per table was designated reporter, recorded the group's suggested 
content-based descriptors, and then wrote th-^ descriptors on a flip chart so that all four 
small groups could see all descriptors. A comprehensive list of grade level descriptors 
was then compiled and distributed to each grade group. Panelists were next asked to 
identify, individually, five or six descriptors that best described what B^sic, Proficient, and 
Advanced students at their grade level should be able to do. Panelists were also asked 
to identify how each descriptor fit the NAEP Writing Framework. Descriptors that did not 
fit within the framework were deleted. The grade-level descriptors for each achievement 
level chosen by a majority of panelists were compiled into a list, resulting in lists of six to 
ten descriptors for each achievement level. The lists of grade level descriptors that 
remained were then discussed by the grade-level groups and suggestions were made for 
modifying wording. Further, panelists were given the option of adding any descriptors 
that had been eliminated if the descriptors fit within the framework and if the panelists felt 
the descriptors were needed. The grade-level groups then reached general agreement 
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that the final lists of descriptors represented what students should be able to do at each 
of the achievement levels. 

During this session panelists also reviewed individually a writing prompt and a set of 
student papers they would not be working with later^ and selected student papers they 
felt were representative of Below Basic, Basic, Proficient, and Advanced writing 
performance. The selections were discussed by the entire group. The purpose of having 
panelists work with these prompts and papers was to familiarize them further with the 
writing prompts and scoring protocols and to help them internalize the relationship 
between their descriptors and the student papers they would be evaluating later. 
Panelists were asked to recommend additional modifications in the descriptions to rectify 
any shortcomings that appeared during this exercise. 

During the evening, ACT'S three content experts reviewed the lists of descriptors for 
consistency with the Writing framework, consistency and logical progression within and 
across grade levels, and editorial quality. The next morning, panelists discussed their lists 
of descriptors and reached general agreement,^ in grade groups, that the lists contained 
skills representative of performance at each achievement level. The major purpose for 
having panelists develop their own set of grade-level specific content-based descriptions 

^ For each grade group, half the panelists worked with about half of the writing 
prompts, half worked with the remaining half of the prompts. At each grade level, 
the writing in response to at least one prompt was reviewed by the group as a 
whole. The prompts were balanced with respect to difficulty and type of writing 
required (narrative, persuasive, informative). 

^ The goal was to have general agreement (or near consensus) on the final lists, not 
a unanimous grade-level approval of each statement. 
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of Basic, Proficient, and Advanced was to ensure that, to the extent possible, all panelists 
would have both a common set of content-based referents to use during the paper 
selection process and a common understanding of student performance for each of the 
three achievement levels at their grade-level. Panelists' final content-based descriptions 
are included later in this report. 

Training in the Angoff Procedure 
Panelists next received approximately one hour of training in the rating process 
modified for the writing assessment to require selection of two student papers to 
represent performance for each achievement level. Emphasis during development of 
achievement level descriptors was on what student performance should be, and panelists 
were now instructed to use their "should-based" descriptions along with other information 
presented to them to evaluate the NAEP Writing prompts and student papers using their 
best judgment of how students at the borderline'* of each achievement level would 
perform on the prompts. The concept of borderline performance was fully explained. 
Panelists were instructed to envision students whose writing performance was borderline 
Basic, students whose performance was borderline Proficient, and students whose 
performance was borderline Advanced. They were asked to think about the level of 

^ Borderline refers to the "cut point" or minimal competency point separating the 
achievement levels. For example, the borderline between Basic and Proficient is 
that point on the NAEP scale, or that level of performance as described in 
panelists' descriptions, that separates Basic from Proficient student performance. 
All students scoring at or above that borderline (but below the borderline for 
Advanced performance) would be classified as Proficient; all students below that 
borderline would be classified as either Basic, or below Basic depending on their 
performance. 
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writing petiormance that might be typical for such groups of students for particular types 
of writing prompts (e.g., persuasive versus informative). For example, what might 
borderline Proficien* students be capable of producing that borderline Basic students 
could not? Panelists were told they would review 40-42 actual student papers, 
representing a roughly equal distribution of the six score points. From these, they would 
select six (6) papers, two for each achievement level, that typified student performance 
at the borderline of ^ach level. Panelists were encouraged to ask questions during this 
training session so that misconceptions or uncertainties could be addressed before 
Round 1 of the "ratingVpaper selection process. 

The Paper Seiection Process 
Panelists completed three rounds of paper selections. Papers, which had already 
been scored on a six-point scale, were randomly mixed, assigned a number, and then 
ordered sequentially. Scores were deleted from the papers and panelists made selections 
without knowing what scores had been assigned to the papers. For Round 1 , panelists 
first reviewed the writing prompt and spent time thinking about how they would write a 
response. They then reviewed the scoring protocols. This process helped ensure that 
panelists would be thoroughly familiar with each prompt and its scoring rubrics before 
selecting papers. Panelists provided six paper selections (two for each achievement level) 
for each prompt and then proceeded to the next prompt, for which the process was 
repeated. During Round 1, panelists used their lists of descriptors and other training 
materials for guidance in the paper selection process. Most panelists completed Round 1 
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in about 5 hours, although some (particularly grade 12 panelists) needed over 8 hours 



i 




for the task. 

Following Round 1, ACT staff entered the panelists' paper selections into a computer 
data base and analyzed the results. The score given each paper was matched in the 
computer data base with the unique prompt and paper number. The mean (or average) 
score for each panelist's selections for each achievement level was computed, and the 
mean for all panelists' selections for each achievement level for each grade group was 
also computed. A computer plot of panelists' individual means for each achievement level 
was produced. 

For Round 2 of the process, panelists were given interjudge consistency information 
using this plot of means. This information allowed panelists to see on the scale where 
the mean of their own paper selections were, relative to the mean for the group and to 
the means for other panelists. Reasons for divergent means, including the possibility that 
some panelists misinterpreted the paper selection task, were discussed. No effort was 
made to coerce panelists to change their selections. Figure 2 is an example of the 
interjudge consistency information shared with panelists. 

For Round 2, panelists were also given student performance data in two forms to 
indicate the "difficulty" of the writing task. One way this information was presented was 
the percentage of students included in the actual NAEP administration who earned 
various scores (1 though 6). Additionally, the "difficulty" information was presented as the 
mean score (1 to 6) for student responses for each prompt for the af^tual NAEP 
administration (Figure 3). Panelists were told that this difficulty information should be 
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How to read: Panelists' mean paper selections were plotted along the horizontal axis (6- 
point scale) as a representation of the panelists' achievement level choice. The letter B 
is the mean for the panelists' Basic level selections, P is their Proficient level mean, and 
A is their Advanced level mean. Individual panelists are identified along the vertical axis 
(e.g., A08F is panelists F in Grade 8, Group A). TTiis panelist's ratings are very near the 
mean for the grade group, for all three achievement levels (shading added). 
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used as a reality check on their paper selections, not as an indication of what their 
specific selections should be. For example, if they had thought two prompts were of 
similar difficulty but the two prompts had very different difficulty values, they were asked 
to reexamine the prompts to determine if they had misinterpreted some aspect of the 
prompt or the difficulty of the writing task, for example. Results of the data analysis 
supported the panelists' own evaluations that the difficulty information was useful but had 
little impact on the panelists' paper selection process. 

The final type of information given to panelists for use during Round 2 (and also 
Round 3) selections was data on intrajudge variability. Each panelist received a list 
containing the mean score for papers they selected for each prompt they reviewed, as 
well as the range of scores for the two papers used to compute the means. Data for 
panelist E, Grade 4 Group B are shown below for illustrative purposes. Panelist E's 
Proficient level paper selection means varied from 1.50 for prompt W10 to 4.00 for 
prompts W5 and W11. This would indicate that panelist E felt prompt W10 represented 
a more difficult writing task than prompt W5, or that he/she misjudged one or more of 
these prompts. For prompt W1 1, both paper selections had to have been scored as 4's 
because the mean paper selection score is 4 and the range of scores is 0. This is an 
indication of consistency in paper selections. For prompt W4, however, the mean paper 
selection score is 3.50 and the range is 3, indicating one paper selected was scored 2 
and one received a score of 5 (5 + 2 ^ 2 = 3.5, the mean; 5 - 2 = 3, the range). This 
is an indication of inconsistency in paper selections. 
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Proficient Level 


Panelist ID 


Prompt 


Mean 


Range 


A04E 


W10 


1.50 


1 




W6 


2.00 


0 




W5 


4.00 


2 




W11 


4.00 


0 




W4 


3.50 


3 


Rater Averages = 




3.00 





Panelists were instructed to review these data and to decide if their initial decisions 
regarding prompt difficulty and paper selections needed to be modified. Panelists were 
pointedly told not to alter their selections solely on the basis of these data; only if they 
believed their initial decisions were flawed or somehow in error. 

For Round 2, panelists reviewed the same sets of prompts and papers they used in 
Round 1 and, using the information on interjudge consistency, intrajudge variability, 
difficulty, and the information provided prior to Round 1, they either confirmed their initial 
paper selections or made new selections to reflect the additional information. About one- 
fourth of Round 1 paper selections were charged during Round 2, most to a paper that 
had received the same score or a paper within one score point of their first selection. 

Following Round 2, panelists' selections v/ere reanalyzed and information concerning 
intrajudge variability was again presented to panelists prior to Round 3. Panelists were 
asked to review each set of papers that appeared to be inconsistent and decide if their 
Round 2 selections accurately reflected their judgments of the students' writing 
performance at the borderline achievement levels. The intrajudge consistency data was 
to be used only to flag papers for reconsideration in the final round of paper selections. 
Again, panelists were told that these data were for their information and that changes in 
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Figure 3 

Sample Feedback on Item Difficulty 

GRADE 4 
Prompt Score Point Distribution 
(% of Students Papers Scored at Each Score Point) 

Group B 



Prompt 


Scores 


Off Task/ 
Omit 


1 


2 


3 


4 


5 


6 


Jimmy Jet 


9.1 


41.1 


28.7 


6.1 


0.8 


0.0 


14.4 


Another Planet 


6.6 


32.0 


35.2 


18.0 


1.6 


0.5 


6.3 


Lunchtime 


3.1 


8.2 


42.4 


30.7 


6.5 


1.3 


8.8 


Magical Balloon 


6.4 


18.7 


40.6 


24.8 


3.5 


0.6 


5.5 


Favorite Object 


5.9 


7.7 


43.5 


26.3 


4.9 


0.7 


11.1 



Mean Scores for Prompts 











Group A 


X 


Group B 


X 


(P) School Year 


2.48 


(P) Jimmy Jet 


2.40 


(N) Dinosaur 


2.89 


(N) Another Planet 


2.76 


(1) Favorite Story 


3.31 


(1) Lunchtime 


3.36 


(P) Spaceship 


2.54 


(N) Magical Balloon 


3.02 


(!) Favorite Object 


3.21 


(!) Favorite Object 


3.21 



P = Persuasive 
N = Narrative 
I = Informational 
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paper selections should be made only if reconsideration of the paper, in its entirety, 
indicated a need to select a different paper. 

For Round 3, panelists reviewed the same set of prompts and papers they used in 
Rounds 1 and 2 using both the new intrajudge variability information and the information 
made available during Rounds 1 and 2. In addition, panelists could discuss, within their 
small groups, selections of papers about which they were unsure. About one-sixth of the 
paper selections were changed during Round 3, with most new selections being within 
one score point of their Round 2 selections. Results from Round 3 are presented in 
Table 2 (see Results section). 

Reyiew of Grade-Lev^ Descriptors and Selection of Exemplar Papers 

On the next day panelists were asked to review sets of student papers for the 
released prompts^ and to select papers that they felt represented typical or average, 
performance for each achievement level for their grade (unlike the paper selection 
process used to determine the numerical results for the NAEP where the emphasis was 
on borderline performance). These papers were then discussed in the small groups 
(table groups) within grades, with each small group reaching general agreement on a set 
of papers (2-3 papers) they felt represented good examples of performance at each of 
the achievement levels. 



^ Educational Testing Services releases, or makes public, several writing prompts 
after each NAEP Writing Assessment. For 1992, these released prompts included 
three for grade 4 and three that were common prompts for grades 8 and 12. 
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These released prompts and possible exemplar papers were further reviewed by a 
group composed of ten teachers from the St. Louis panel and ten writing experts 
nominated by personnel in state departments of education, officers of the National Writing 
Project, and officers of the National Council of Teachers of English at a meeting in 
Orlando, Florida September 4-6, 1992. The final set of prompts and papers selected by 
this group are included in Appendix D. After the Public Comment Forum process has 
been completed, ACT will recommend a final set of exemplar papers to NAGB for use in 
reporting the 1992 NAEP Writing results. 

For the achievement level descriptions, panelists were asked to review the other two 
grade groups' descriptions to determine whether they were consistent with their grade 
group descriptions and to suggest revisions. They were also asked to review once again 
their grade group descriptions to determine whether any wording changes should be 
made. The result was three sets of achievement level descriptions (one for each grade 
group) generally acceptable to all three grade groups. These descriptions are included 
in the Results section following. These descriptions were also reviewed and further 
revisions were recommended at the Orlando meeting. The Orlando group's descriptions 
are included in Appendix D. 

Results 
Numerical Results 

Table 2 summarizes the numerical results produced by the panelists. The mean score 
(e.g., 2.65) is the average paper score across prompts, across panelists. The standard 
deviation (abbreviated SD) is a measure of the amount of variation across panelists in 
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Table 2 

Mean Writing Achievement Levels 
By Rounds 



GRADE 4 (22 Panelists, 9 Prompts) 





Borderline 


Borderline 


Borderline 


Round 


Basic 


Proficient 


Advanced 


1 


2.65 


3.64 


4.38 




(SD = .74) 


(SD = .47) 


(SD=.31) 


2 


2.67 


3.65 


4.67 




(SD = .50) 


(SD = .48) 


(SD = .20) 


3 


2.47 


3.66 


4.72 




(SD = .38) 


(SD = .35) 


(SD = .20) 


GRADE 8 (21 Panelists, 1 1 Prompts) 




Borderline 


Borderline 


Borderline 


Round 


Basic 


Proficient 


Advanced 


1 


2.75 


3.89 


4.51 




(SD=.50) 


(SD = .42) 


(SO =.29) 


2 


2.74 


3.66 


4.57 




(SD = .36) 


(SO = .49) 


(£D = .33) 


3 


2.64 


3.70 


4.58 




(SD = .44) 


(SD = .40) 


(SD = .31) 


GRADE 12 (23 Panelists, 12 Prompts) 




Borderline 


Borderline 


Borderline 


Round 


Basic 


Proficient 


Advanced 


1 


3.04 


4.00 


4.56 




(SD=.65) 


(SD = .59) 


(SD = .39) 


2 


2.89 


3.92 


4.78 




(SD = .76) 


(SD = .59) 


(SD = .30) 


3 


2.80 


3.89 


4.90 




(SD = .49) 


(SD = .48) 


(SO = .27) 



Description of NAEP Writing Achievement Levels-Setting Process and Proposed Achievement l-evel Definitions 



P^ge 19 



mean scores of papers selected. In general, this variation declined substantially from 
Round 1 to Round 3, indicating a move toward greater consensus among panelists. 
Because panelists' mean paper selection scores cannot be converted into NAEP scale 
scores at this time,^ the only way in which panelists' numerical results can be evaluated 
is by comparing means across achievement levels within grades for all prompts, and 
across grades for prompts that are common to more than one grade. The results across 
achievement levels within grades appear to be logical and consistent, with the mean 
score higher for borderline Advanced than for borderline Proficient and higher for 
borderline Proficient than for borderline Basic (Table 2). 

Because papers written in response to prompts common to more than one grade are 
scored using common scoring protocols (e.g., the same scoring protocol was used for 
the prompt "Embarrassing Incident" which was administered to both Grades 8 and 12), 
comparisons of mean paper selection scores across grades is possible for these 
prompts. That is, if student writing performance should improve from grade 4 to grade 
8, and from grade 8 to grade 12, then mean paper selection scores vithin achievement 
levels across grades should be higher for grade 8 than grade 4, and higher for grade 12 
than grade 8. As can be seen in Tables 3A-3B this was the case for all three 
achievement levels for grades 4 and 8, and for grades 8 and 12. Figures 4A and 4B 
present a graphical depiction of these results. 



^ Achievement levels on the NAEP scale cannot be calculated and reported until 
Educational Testing Service completes data analysis on the 1992 NAEP Writing 
Assessment results later this year. 
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Table 3A 



Mean Round 3 Writing Achievement Levels for 
the 3 Common Grade (4th & 8th) Items 





Borderline 


Borderline 


Borderline 


Grade 


Basic 


Proficient 


Advanced 


4th 


2.35 


3.56 


4.51 


(n=22) 


(SD = .39) 


(SD=.40) 


(SD=.25) 


8th 


3.07 


3.93 


4.87 


(n=21) 


(SD = .50) 


(SD = .67) 


(SD = .45) 



Table 3B 



Mean Round 3 Writing Achievement Levels for 
the 7 Common Grade (8th & 12th) Items 





Borderline 


Borderline 


Borderline 


Grade 


Basic 


Proficient 


Advanced 


8th 


2.46 


3.65 


4.42 


(n=21) 


(SD = .55) 


(SD = .39) 


(SD = .35) 


12th 


2.88 


4.03 


4.94 


(n=23) 


(SD = .44) 


(SD = .55) 


(SD = .32) 
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Rgure 4A 



Grade 4 v. Grade 8 Common Items 



* Advanced Level 
■ Proficient Level 

• Basic Level 



CO 

o 

a. 4 

o 
o 

O 3 

0^ 



04 



08 



GRADES 



ERIC 



Datcription of I^AEP Writing Achi«v«m«nt lj*v«ls-S«tting ProctM and PropOMd Achi«v*m*m |jiv*l Definitions 



Pag* 22 



Figure 4B 



Grade 8 v. Grade 12 Common Items 
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Achievement Levels Descriptions^ 
m4th-Grade Writing Achievement Level Descriptions 
BASIC Performance: 

• Basic writing at the fourtli-grade level should state or clearly impiy a central idea 
with some supporting details. 

• Basic writing should be organized in a sequence of ideas that makes sense. 

• There should be some evidence of a beginning (introduction), middle (body), and 
ending (conclusion). 

• Word choice is simple and vocabulary is general. 

• The writing shows some evidence of an appropriate response to the writing task. 

• The writer should use complete sentences, and some evidence of correct 
punctuation, capitalization, spelling, and grammar should be demonstrated. 

PROFICIENT Performance: 

• Proficient writing at the fourth-grade level should state or clearly imply a central 
idea, expressed with enough detail to communicate the purpose to the intended 
audience. 

• Proficient writing should be organized logically through a beginning (introduction), 
middle (body), and end (conclusion). 

• The writer uses descriptive language to clarify and enhance ideas. 

• The form and content of the writing responds to the writing task. 

• Vocabulary choices begin to enhance details. 



^ These descriptions were reviewed and extensive revisions suggested in a meeting 
in Orlando, Florida September 4-6. Revised descriptions and the descriptions 
presented above will be evaluated for appropriateness prior to recommendation 
by ACT to the Board. Public comment on the merits of the two sets of 
descriptions are particularly welcomed. 

ErJc . 3 
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^ • The writer should be able to use complete and varied sentences that include few 
errors in capitalization, commas, ending punctuation, grammar, and spelling. 

ADVANCED Performance: 

• Advanced writing at the fourth-grade level should clearly state a central idea and 
elaborate on that idea with descriptive and supportive details, including illustration 
examples, and analogies. 

• The form and content of the writing responds to the writing task. 

• Organization should progress logically through a beginning (introduction), middle 
(body), and end (conclusion), showing evidence of purpose and recoqnition of 
audience. 

• Ideas flow smoothly within and between paragraphs. 

• There should be evidence of creative, abstract thought. 

• Vocabulary should be thoughtfully selected for illustration and elaboration. 

• Sentence structure should be varied and include complex sentences with few or 
no errors in spelling, capitalization, punctuation, and grammar. 

m8th-Grade Writing Achievement Level Descriptions 

BmO:C Performance: 

• Basic writing at the eighth-grade level should display the writer's knowledge of 
simple sentence and paragraph structure. 

• The writing should be an appropriate response to the writing task in its form and 
content. 

• The writer should be able to use basic vocabulary effectively. 

• The content should be understandable. 

• The writer should be able to maintain minimal focus. 

• The writing should demonstrate some awareness of audience. 
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• The writer should be able to use correct capitalization and punctuation of a simple 
sentence. 

• The writer should be able to use supporting details. 

• There should be some evidence of organizational skills. 
PROFICIENT Performance: 

• Proficient writing at the eighth-grade level should display organization that is both 
observable and functional with an effective beginning, middle, and ending. 

• The writer should be able to use a variety of sentence structures. 

• The sequence of ideas should be clearly understandable. 

• The writing should demonstrate awareness of audience and purpose. 

• The topic should be clearly stated and supported by details. 

• The writer should be able to construct a paragraph within which most sentences 
support the central topic of that paragraph. 

• The writer should demonstrate correct mechanics and usage. 

• The writing should include some effective transitional elements. 
ADVANCED Performance: 

• Advanced writing at the eighth-grade level should be an extended, appropriate 
response to the writing assignment. 

• Supporting details should be varied and elaborated. 

• The writing should be clear and cohesive, and should maintain clear focus. 

• The writing should include a well-crafted beginning, middle, and ending. 

• The writer should be able to use varied sentences to express expanded ideas. 

• The writer should be able to express analytical, critical, and/or creative thinking. 

• The writer should begin to develop a personal style or voice. 
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• The writer should be able to use transitional elements proficiently. 

• The writer should be able to use correct punctuation consistently. 

• The writer should be able to demonstrate superior use of vocabulary/language. 

• Writers should be able to use a variety of strategies such as analogies, 
illustrations, examples, anecdotes, and/or figurative language. 

mlZth-Grade Writing Achievement Levei Descriptions 

BASIC Performance: 

• Basic writing the twelfth-grade level should demonstrate appropriate response in 
form, content, and language to all parts of the writing task. 

• Basic writing should demonstrate supporting details. 

• Writing should show some evidence of organization. 

• Basic writing should demonstrate basic knowledge of grammar, usage and 
mechanics. 

PROFlCiENT Performance: 

• Proficient writing at the twelfth-grade level should be an effective response in its form, 
content, and language to all parts of writing tasks. 

« Proficient writing should demonstrate illustrative and supporting details. 

• Proficient writing should demonstrate organizational skills through a logical sequence 
and flow of ideas. 

• Writing should demonstrate appropriate language/vocabulary for a variety of 
audiences and purposes. 

• Writing should demonstrate clarity of thought through the use of clear, concrete 
language. 

• Writing should evidence analytical/critiCal/creative thinking skills. 

• Proficient writing should demonstrate knowledge of grammar, usage, and mechanics. 
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ADVANCED Performance: 

• Advanced writing at the twelfth-grade level should respond effectively and fully in form, 
content, and language to all parts of the writing task. 

• Advanced writing should use rich and varied illustrative and supportive details. 

• Advanced writing should demonstrate organizational skills appropriate to the task, 
resulting in fully developed ideas. 

• Advanced writing should engage readers through powerful, compelling, and rich 
language and through facility with the nuances of language. 

• Writing should evidence a variety of sentence structures and lengths. 

• Advanced writing should show some sophistication of analytical/critical/creative 
thinking skills. 

• Writing should include a variety of strategies (i.e. anecdotes, repetition, literary 
devices, comparison/contrast) to support and develop ideas with a clear structure. 

• Advanced writing should display sophisticated transitional devices (i.e. transitional 
phrases, sentences, and paragraphs). 

• Advanced writing should demonstrate a command of grammar, mechanics, and 
usage. 

Panelists' Evaluations 

Panelists completed four evaluation questionnaires, one at the end of each day. To 
the extent possible, the questionnaires replicated the questions used in the 1990 
mathematics achievement levels-setting process and the 1992 mathematics and reading 
achievement levels-setting processes. 

In general, panelists rated the experience as very positive, their understanding of the 
tasks thvjy were performing as high, their understanding of the definitions of the 
achievement levels as very clear, their conceptions of students at the borderline of the 
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achievement levels as well formed, and the time for performing tasks and the assistance 
received from ACT staff as just about right. A brief description of some of the more 
important results is presented below. A full description and analysis of panelists' 
evaluations will be available at a later date. 

Understanding {^Achievement Level Descriptions 

Panelists were asked to indicate their level of understanding of the description of 
student performance at each achievement level, at the end of the first day and following 
each round of "ratingsVpaper selections. Response options ranged from 1 =not at all 
clear to 5= absolutely clear. Responses for the Basic achievement level are typical of 
responses to this item, with the number of panelists rating this item as more than 
somewhat clear increasing from 49 after Day 1 to 59 after Round 3. Similar increases 
were observed for the Proficient and Advanced levels. Figure 5 provides an example of 
the change in panelists* understanding of the achievement levels. At the end of Day 1 , 
only 21 percent of the panelists felt the definition of Basic was absolutely clear; by the end 
of Round 3, 36 percent felt the definition was absolutely clear. 

Conception of Borderline Achievement Level 

At the end of the first day and following each round of "ratings'Vpaper selections, 
panelists were asked to indicate how well formed their conception of borderline 
performance was at each achievement level. Response options ranged from 1 =not well 
formed to 5 = very well formed. For borderline Basic performance, the number of panelists 
rating this item as more than somewhat well formed doubled from 26 after Day 1 to 52 
after Round 3. Similar results were observed for the Proficient and Advanced levels. 
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Rgure 5 

Example of Improvement in Panelists' Understanding, 
End of Day 1 to End of Round 3 



At the time I selected the XXX set of papers, my understanding 
of the definition of student performance at the Basic level of 

achievement was: 



100 J 

90 -- 

80 - 
70 

60 -- 

50 -- 

40 -- 

30 -- 

20 -- 
10 
0 



21 



53 



26 



50 



18 



26 



53 




End of Day 1 



Round 1 



Ms 1114 Ds 



Round 3 



1 Not At All Clear 
2 

3 Somewhat Clear 
4 

5 Absolutely Clear 



Note: Day 1 » Level of Understanding 



*> ^ 
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Figure 6 provides an example of the change in panelists' conceptions of borderline 
performance. At the end of Day 1, only 14 percent of the panelists felt their conception 
of borderline Basic performance was very well formed; by the end of Round 3, 30 percent 
felt their conception was very well formed. 

Confidence in Paper Selections . Panelists were asked, at the end of each round of 
ratings, to indicate their level of confidence in the paper selections they provided. 
Response options ranged from 1 =not at all confident to 5=totally confident. The number 
of panelists indicating they were more than somewhat confident in their selections 
increased from 19 at the end of Round 1 to 57 at the end of Round 3. Figure 7 shows 
the change in panelists' confidence levels from Round 1 to Round 3. At the end of 
Round 1, only 3 percent of the panelists were totally confident in selections; 71 percent 
were at least somewhat confident (3 panelistS"4.6%-did not respond). At the end of 
Round 3, 26 percent were totally confident, and 97 percent (100% of those responding) 
were at least somewhat confident. 
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Rgure 6 

Example of Improvement in Panelists' Conception, 
End of Day 1 to End of Round 3 



At the time I selected the XXX set of papers, my conception of 
Borderline Basic performance was: 

100 -r 
90 -- 
80 -- 
70 -- 



60 -- 




End of Day 1 Round 1 Round 3 



Els M4 03 



1 Not Well Formed 
2 

3 Moderately Well Fomied 
4 

5 Very Well Formed 



Note: Day 1 » Level of Understanding 



OP 
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Rgure 7 



Panelists' Confidence in Paper Selections 



Tha most accurate description of my level of confidence In the 
papers I selected to represent the three achievement levels 
during the XXX rating session Is that i was: 

100 -r 
90 -- 
80 -- 



70 -- 




Round 1 Round 2 Round 3 



■ 5 1]I]4 DS 



1 Not At All Confident 
2 

3 Somewhat Confident 
4 

5 Totally Confident 
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Section 1 -Identification and Selection of Panelists 
mSampling Plan For NAGB Advisory Panels 

NAGB has indicated that the Achievement Levels Panels "shall be composed of 70 
percent educators and 30 percent non-educators overall, and reflect a balance of gender, 
race/ethnicity, and geographic location" (NAGB, 1991, p. 13). NAGB has further 
stipulated that classroom teachers should comprise 55 percent of the group. ACT 
believes that all prospective panelists (educators and non-educators) should be familiar 
with the knowledge and skills required by the content area panel for which they are 
selected and with the grade level group to which they would be assigned. Such 
comparable, relevant background experiences will bolster the validity of the process and 
would likely ensure greater group cohesiveness. Further, it is ACT'S belief that a 
consensus formed from a panel of persons (meeting the stated distributional requirements 
of NAGB policy) having practical experience with and knowledge of students at the 
specified grade levels will best meet the intent of NAGB policy calling for a "consensus 
among a broadly representative" sample. 

Because there are many stakeholders involved in this process--those with great and 
sincere interest in the outcomes of this effort-we believe it is important to have their input 
into this critical aspect of the process. To that end, ACT distributed a document detailing 
our panelist selection design to national organizations and to other groups that have an 
important role in education or in the different content areas. We then met with 
representatives of those groups during the week of January 6, 1992 to discuss concerns 
about, or objections to the procedure, when any were raised. 

There will be 9 panels, one for each combination of test and grade level. Each panel will 
contain 20 primary members and 2 backup members. Approximately 1 1 panelists will be 
teachers, 3 will be non-teacher educators, and 6 will represent the general public, on 
average, for each grade level. The 2 backup members are intended to increase the 
likelihood that there will be at least 20 panelists for standard setting in each content area 
and grade level. The backup members will participate fully in the deliberations; therefore, 
if all 20 primary members attend the standard setting meetings, then as many as 22 
panelists could participate in the process. 

Across the three grade-level panels for each content area, the NAGB requirement of 55 
percent classroom teachers, 15 percent non-teacher educators, and 30 percent general 
public (non-educators) will be met. 

General strategy for selecting panel members 

One way to characterize strategies for selecting panelists is according to the breadth of 
the populations from which the panelists are selected. In the ^^political" approach, we 
would have identified the professional and political organizations that have the greatest 
interest and stake in the panels' deliberations, then ask these organizations to nominate 
panelists. Under this method, the populations sampled would have expertise and 
influence in setting educational policy, but would be fairly narrowly defined from the 
perspective of educational practice. In the "sampling" approach chosen, ACT will select 
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districts from which individuals in pre-specified positions or occupations will be identified 
as nominators. The nominators will be asked to nominate panelists, according to fixed 
criteria, from an identified group of practitioners. ACT will then select panelists from the 
pool of nominees. Under this method, panelists would be more representative of 
practitioners, but would be potentially less influential in setting educational policy, than 
under the political approach. 

After review both by ACTs internal Technical Advisory Team (TAT) and the Technical 
Advisory Committee for Standard Setting (TACSS) of the advantages and disadvantages 
of each strategy, ACT has decided to recommend a variation on the sampling approach. 
As indicated above, we believe the best strategy for arriving at a consensus among a 
broadly representative set of panelists is to follow, as closely as possible, procedures 
called for in sampling methodology. The discussion that follows applies to selecting 
panelists for each content area (Mathematics, Reading, Writing). Separate samples of 
districts will be selected for each content area and for each category of panelists 
(teachers, non-teacher educators, and individuals in the general public-nine samples in 
ail). Once the nominators have been identified by name for each sampled district, names 
of the nominators will be sent to organizations such as the International Reading 
Association, the National Council of Teachers of Mathematics, and the National Council 
of Teachers of English. In districts where members of the organizations are active, the 
organizations may contact a listing of nominators and urge their participation. 

Defining target panelist groups 

In order to select panelists by the sampling method, we have been as precise as possible 
in defining the target groups from which panelists will be selected. (Please refer to the 
Nomination Forms in Appendix B.) The following definitions will be used for the target 
panelist groups: 

Teachers . Panelists nominated to the pool must meet all of the following qualifications: 

a. At least five years of overall teaching experience. 

b. At least two years of teaching experience in the subject area (writing, 
mathematics, or reading) and in the indicated grade (4, 8, or 12). This criterion 
will need to be modified at the 8th and 12th grade levels for reading and~to some 
extent-for writing, because most schools do not have reading and writing 
teachers, perse, at those grade levels. In these cases, panelists will have to have 
at least two years of teaching experience in a subject area in which reading 
(writing) is an extremely important skill, such as English, social studies and 
literature. 

c. Judged to be "outstanding" in their professional performance by a supervisor or 
someone in a position to make that judgment. 

Non-teacher educators . Panelists will be nominated by two groups of nominators: 
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a. Non-teacher educational staff at primary and secondary educational institutions 
(e.g., counselor, principal, or curriculum supervisor), in District offices, and in State 
Departments (e.g., Assessment Directors). 

b. Professors or administrators at postsecondary institutions. 

Panelists nominated from eitlner non-teaclier educator group must have familiarity and 
professional experience with the subject matter of the test at the indicated grade level, 
and must be judged "outstanding" in their professional performance by the nominator. 
T . nominator will also be asked to indicate the reason for which the person is 
considered to be outstanding. 

G- (eneral public . Persons nominated from the general public to be panelists must have 
familiari^' with the subject matter at the indicated grade, and must not currently be 
employe by an educational institution. In addition, nominators will be discouraged from 
nominati. \g fornoer teachers and educators, as these groups will have been targeted for 
nomination by other groups of nominators. For example, a parent of a fourth-grade 
jdent and an employer of recent high school graduates might qualify as members of 
the general public target population, but a teacher, principal, or district superintendent 
would not qualify. 

Representativeness of panels 

This sampling methodology is intended to achieve "broad representativeness" on the 
panels by sampling from among public school districts and private schools to identify 
nominators. The districts will be selected according to region of the country, affiliation 
with public/private institutions, type of community (socio-economic status of district 
residents), and affiliation with large/not large enrollment school districts. We will achieve 
representativeness with respect to region, public/private, type of community, and size of 
districts by using these variables to select stratified samples of districts. Each nominator 
will be asked to nominate one or more individuals in the relevant target group of panelists, 
and to report the sex and race/ethnicity of each nominee. Panelists will then be selected 
from the pool of nominees so as to maximize the balance of gender and race/ethnicity, 
as the primary considerations, and geographical region, school affiliation (public/private), 
type of community (socio-economic status), and district enrollment size, as secondary 
considerations (each of equal weight). While the methodology does not assure 
proportional representation among these criteria, it does assure diversity among the 
members selected for the panels. We believe that the interests and concerns of the 
individuals that will come to serve as panelists will be representative of a broad cross- 
section of citizen-consumers of education in the U.S. today. 

Technical considerations 

While this method of selecting a sample of districts represents "probability sampling" in 
which each member of a well-defined target population has a known, positive probability 
of being selected, the selection of panelists, as just described, is not "probability 
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sampling," per se. Probability sampling at the panelist level is not possible because of the 
unknown and subjective judgments of the nominators. By using aspects of sampling 
methodology, however, we will be able to select broadly representative panels through 
which diverse points of view can be expressed. 

We will obtain a large pool of nominees from which we can select panels. It will not be 
possible, given the small panel sizes, to ensure that each panel is representative with 
respect to each combination of characteristics (e.g., black females in large districts in the 
western U. S.). Moreover, it will not be possible to ensure proportional representation of 
categories. We will, however, attempt to select panels so that there is at least one 
panelist for each category, and so that the most equitable possible balance of gender and 
race/ethnicity is met. 

Definitions of stratification variables 

Individuals from the following classification categories to be represented in the samples: 
Region: Four NAEP regions (West, Central, Southeast, Northeast) 

Public/private: Use the MDR* classification. 

Community type: MDR Orshansky indicator of SES (type 1 = 25 percent or more 
district residents below federal poverty level; type 2 = all other 
districts). 

District size: MDR district enrollment data (large = enrollment > 50,000 
students; not large = enrollment < 50,000 students). 

* The name MDR refers to a computer file of school information maintained by Market 
Data Retrieval, Inc., of Westport, Connecticut. 

Sample for teacher panelists 

The district superintendent and the president/head of a teacher's group will be asked to 
nominate teachers from public school districts and principals or superintendents of private 
schools will be asked to nominate teachers from those schools. The sample of districts 
will be stratified by region, community type, and student enrollment size. The sample of 
private school principals will be stratified by region only. 

Each district superintendent, association president, and private school principal or 
superintendent will be asked to nominate from one to four individuals from each of the 
three grade levels from among the teachers in his or her district or school who meet the 
criteria for nomination. The nominators will also be asked to keep in mind the need for 
appropriate distributions of sex and ethnicity when making their selections, and to report 
the sex and ethnicity of each of their nominees. Finally, these nominators will be asked 
to permit (or secure permission for) any nominees who are selected as panelists to attend 
the achievement levels setting meetings. (See the sample letter to teacher nominators 
in Appendix B.) 
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Sample for non-teacher educator panelists 

The nominators of the non-teacher educators will themselves be non-teacher educators, 
as previously defined. The nominators may nominate themselves or any of their 
colleagues who meet the specified requirements. 

The nominators for the primary and secondary school district level and state level non- 
teacher educators will be selected from the "MDR Personnel File" according to job title. 
Job titles included in the sample are listed in Table 1. The nominators for the 
postsecondary non-teacher educators will consist of Deans of Liberal Arts, Humanities, 
or Education at two-year and four-year postsecondary institutions from the 1992 Higher 
Education Directory, published by Higher Education Publications, Inc. 

The nominators in both the primary /secondary district/state level sample and the 
postsecondary school sample will be requested to nominate from one to three individuals 
representing each of the three grade levels in a particular subject area depending on 
district/school/college size. They will also be requested to keep in mind the distribution 
of sex and ethnicity among their colleagues and to provide information on these variables 
for the people they nominate. (See the sample letter to nominators of non-teacher 
educators in Appendix B.) 

General public sample 

Nominations for the general public panelists will be obtained from these different groups: 
1) the Chair of the Education Committee of the local Chambers of Commerce, 2) mayors 
(or equivalent) of local municipalities, and 3) chairs of public and private school boards. 
The nominators will be asked to nominate one to four individuals for each of the three 
grade levels. Nominations from these three groups of nominators may include 
themselves, if they meet the criteria, but the general public category is not restricted to 
members of the Chamber of Commerce, mayors, nor school board chairs; it does 
exclude the educational community. In addition, nominators will specifically be 
discouraged from nominating former teachers and educators, in order to ensure that this 
sample does represent the non-educational community. (See the sample letter in 
Appendix E to this group of nominators.) Each nominator will also be requested to 
provide information on the sex and race/ethnicity of the individuals nominated. 
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Table 1 

Job Titles for Primary and Secondary School 
Non-Teacher Educator Sample 

K-12 Curricular/lnstructional Supervisor 

Elementary Curricular/lnstructional Supervisor 

Secondary Curricular/lnstructional Supervisor 

Vocational Education Supervisor 

K-12 Carer Education Supervisor 

Elementary Career Education Supervisor 

Secondary Career Education Supervisor 

K-12 Guidance Counselor/Supervisor 

Elementary Guidance Counselor/Supervisor 

Secondary Guidance Counselor/Supervisor 

K-12 Social Studies Supervisor 

Elementary Social Studies Supervisor 

Secondary Social Studies Supervisor 

K-12 Math Supervisor 

Elementary Math Supervisor 

Secondary Math Supervisor 

K-12 English/Language Arts Supervisor 

Elementary English/Language Arts Supervisor 

Secondary English/Language Arts Supervisor 

K-12 Reading Supervisor 

Elementary Reading Supervisor 

Secondary Reading Supervisor 

School Board President 

Teacher Personnel Supervisor 

Academic Testing Supervisor 

Affirmative Action Supervisor 

Elementary Principal 

Secondary Principal 

Assistant Principal 

Admissions Director 

Guidance Counselor 

Guidance/Testing 

Guidance/College Advisor 

Guidance/Vocational Advisor 

School Psychologist 

State Commissioner of Education 
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ACT will use the methodology previously described to draw a sample of districts for 
identifying persons to nominate panelists from the general public. Names and addresses 
will be obtained from printed reference materials, when available, and by telephoning 
elected public officials (mayors) and school district offices when necessary. The names 
and addresses of mayors for the municipalities represented by school districts sampled 
will be obtained from the Municipal Yearbook, for example. And, names and addresses 
of presidents/chairs of district school boards wili be obtained through the Directory of the 
National Association of School Boards. 

Sample size 

In order to obtain a sufficiently large pool of nominees from which to select panelists, we 
will have to sample many nominators. To determine appropriate sample sizes for the 
nominators, we made assumptions about: 

a. the participation rate of the nominators 

b. the number of individuals nominated by nominators 

c. the distribution of sex and ethnicity among the nominees, and 

d. the willingness of nominees to serve as panelists. 

Our assumptions were that: 

a. Approximately two-thirds of superintendents, heads of teacher's associations, and 
private school principals or superintendents will participate. Approximately one- 
half of the non-teacher educator nominators will participate. Approximately one- 
half of the general public nominators will participate. 

b. Superintendents, heads of teacher's associations, and private school principals 
or superintendents will nominate an average of 2.0 panelists per grade level, for 
a total of 6 panelists. Non-teacher educator nominators will nominate an average 
of 1.5 panelists per grade level, for a total of 4.5 panelists. General public 
nominators will nominate an average of 1 .5 panelists per grade level, for a total 
of 4.5 panelists. 

c. Superintendents, heads of teacher's associations, and heads of private schools 
will nominate females and males in roughly a two-to-one ratio, and 
Caucasian /whites and minorities in roughly a five-to-one ratio. The other two 
groups will nominate females and male;: in roughly equal proportions, and 
Caucasian /whites and minorities in roughly a seven-to-one ratio. 

d. Approximately three-fourths of the teacher nominees will participate. 
Approximately one-half of the non-teacher educators nominated will participate. 
Approximately one-third of the general public nominees will participate. 

Using these assumptions, we determined that the following number of nominators are 
needed for each category of panelists: 
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1. Teachers: a total of 60 nominators for each content area; 

approximately 85 percent of the nominators will be 
superintendents and heads of teacher's associations (in 
equal numbers) and 15 percent will be private school 
superintendents/principals/head masters 

2. Non-teacher educators: a total of 40 nominators for each content area; 

approximately half of the nominators will be from each of 
the two groups 

3. General public: a total of 80 nominators for each content area; 

approximately one-third of the nominators will be from each 
of the three groups 

Documentation 

Nominees who agree to serve on the panels will be required to submit appropriate 
documentation of their membership in the relevant target population. Nominees who do 
not provide appropriate documentation will be eliminated from the selection process. 



Appendix B 
Agenda 



AGENDA 



Writing Actiievement Levels-Setting Meeting 
July 24-28, 1992 



Friday. July 24 

Check-In 
5:00-6:00 PM 

6:00-7:00 PM 

7:00-8:00 PM 

8:00-9:00 PM 



Ritz-Carlton 

Registration, Get Acquainted, and Cash Bar 
(outside the Plaza Room) 

Welcome Dinner 
(Plaza Room) 

Orientation, Review of Agenda 
(Plaza Room) 

Description of NAGB and NAEP 
(Amphitheater) 



9:00 PM 



Adjourn 



Saturday. July 25 

8:00-8:30 AM 

8:30-10:00 AM 

10:00-10:15 AM 

10:15-11:15 AM 
11:15-Noon 
Noon-1:00 PM 



Continental Breakfast 
(outside the Amphitheater) 

Description of the Achievement Levels-Setting Process; Definitions 

of Achievement Levels. 

(Amphitheater) 

Break 

(outside the Directors Room) 
Take NAEP Exam 
Discuss Scoring Guides 
Lunch 



1:00-3:00 PM 
3:00-3:15 PM 
3:15-4:30 PM 
4:30-5:00 PM 

5:00 PM 

Sunday, July 26 

8:00-8:30 AM 

8:30-8:45 AM 
8:45-9:45 AM 

9:45-10:00 AM 

10:00-Noon 

Noon-1:00 PM 
1:00-5:00 PM 
5:00 PM 



Develop descriptors and operationalize definitions of achievement 
levels. 

Break 

(outside the Directors Room) 

Develop list of grade level descriptors and definitions of 
achievement levels. 

Meet with panelists from all grade groups to discuss and critique 

definitions. 

(Amphitheater) 

Evaluation of Day's Activities 
(Amphitheater) 



Continental Breakfast 
(outside the Amphitheater) 

Orientation of Day's Activities 
(Amphitheater) 

Discuss any issues or concerns with definitions resulting from 
Saturday's sessions. Finalize grade level definitions to be used in 
ratings. 

Break 

(outside the Directors Room) 

Training in Angoff Rating Procedure (with practice items) including 
descriptions of each round of ratings. 

Lunch 

Round 1 Rating (Breaks when needed) 
Evaluation of Day's Activities 



Monday. July 27 



8:00-8:30 AM Continental Breakfast 

(outside the Amphitheater) 

8:30-9:30 AM Discussion of Results from Round 1 Ratings. Instructions for 

Round 2 Ratings. 
(Amphitheater) 

9:30-10:00 AM Grade groups questions and answers on Round 1 and for Round 

2 Ratings. 

10:00-12:30 PM Round 2 Ratings (Breaks when needed) 

12:30-1:30 PM Lunch 

(Plaza Room) 

1 :30-2:30 PM Discussion of Results from Round 2 Ratings. Instructions for 

Round 3 Ratings. 

2:30-5:00 PM Round 3 Ratings (Breaks when needed). 

5:00 PM Evaluation of Day's Activities 



Tuesday. July 28 

8:00-8:30 AM 

8:30-9:00 AM 
9:00-9:30 AM 
9:30-10:30 AM 

10:30-10:45 AM 

10:45-Noon 
Noon-1 :00 PM 
1 :00 PM 



Continental Breakfast 
(outside the Amphitheater) 

Special Ratings 

Review of ALS Process 

Review of grade level achievement levels descriptions and sample 
items. 

Break 

(outside the Directors Room) 

Wrap-up Review and Evaluation of Entire Process 

Lunch 

Adjournment 
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GENERIC DEFINITIONS 



Basic 

This level, below proficient, denotes partial mastery of knowledge and skills that are 
fundamental for proficient work at each grade~4, 8, and 12, For 12th grade this will be 
higher than minimum competency skills (which normally are taught in elementary and 
junior high schools) and will cover significant elements of standard high school-level work. 



Proficient 

This central level represents solid academic performance for each grade tested-4, 8, and 
12. It will reflect a consensus that students reaching this level have demonstrated 
competency over challenging subject matter and are well prepared for the next level of 
schooling. At grade 12 the proficient level will encompass a body of subject-matter 
knowledge and analytical skills, of cultural literacy and insight, that all high school 
graduates should have for democratic citizenship, responsible adulthood, and productive 
work. 



Advanced 

This higher level signifies superior performance beyond proficient grade-level mastery at 
grades 4, 8, and 12. For 12th grade the advanced level will show readiness for rigorous 
college courses, advanced technical training, or employment requiring advanced 
academic achievement. As data become available, it may be based in part on 
international comparisons of academic achievement and may also be related to Advanced 
Placement and other college placement exams. 



Appendix D 



Achievement Level Descriptions 
Representing Student Writing Performance as Developed 
by the Orlando Content Validity Study Group 

September 4-6, 1992 



1992 NAEP Writing Achievement Levels Descriptions 
Orlando Content Validity Meeting 
September 4-6, 1992 

The prevailing desire in this group was to develop descriptions of grade level performance 
within achievement levels rather than to develop descriptions of achievement levels within 
each grade level. Although the objective is to develop descriptions of achievement levels 
within grade level, these descriptions are presented here for review and comment. 

BASIC PERFORMANCE IN WRITING 
GRADE 4 

Basic writing at the fourth grade level should: 

• state or clearly imply a central idea 

• demonstrate some evidence of an appropriate response to a writing task 

• have a begiiming (introduction), middle (body), and ending (conclusion) 

• include some supporting details 

• use clear language 

• contain complete sentences 

• contain few errors in punctuation, capitalization, spelling, and grammar that may 
interfere with communication 

GRADE 8 

In addition to fourth grade expectations, Basic performance in writing at the eighth grade 
level should: 

• maintain consistent focus 

• respond appropriately to the task 

• demonstrate organization appropriate to the writing task 

• use supporting details 

• contain complete sentences and paragraphs 

GRADE 12 

In addition to fourth and eighth grade expectations, Basic performance in writing at the 
twelfth grade should: 

• reveal personal style or voice 

• demonstrate reflection and insight 



PROFICIENT PERFORMANCE IN WRITING 



Given the context of a timed testing situation with a predetermined topic and given the age 
and grade level of the student, writing at the Proficient level should clearly respond to the 
writing task, including effective use of organization, elaboration and the conventions of 
written English, 

GRADE 4 

At the fourth grade level, student writing should include: 

• a clearly stated or implied central idea 

• an awareness of the intended audience 

• a logical organization appropriate to the task 

• sufficient elaboration to clarify and enhance the central idea 

• language appropriate to the task and intended audience 

GRADES 

In addition to fourth grade expectations. Proficient performance in writing at the eighth 
grade should include: 

• a clearly stated or implied central idea 

• a demonstrated awareness of the intended audience 

• a logical and observable organization (e.g., signaled by transitional elements, 
paragraphing) 

• sufficient elaboration to clarify and enhance the central idea 

• language appropriate to the task and intended audience 

GRADE 12 

In addition to jourth and eighth grade expectations. Proficient performance in writing at the 
twelfth grade should include: 

• a clearly stated or implied central idea which responds fully to the writing task, and 
may evidence analytical, critical or evaluative thinking 

• clarity of thought and appropriate language for the intended audience 

• a logical organization which includes sequencing and transitional elements 

• convincing elaboration and development which enhances the central idea 



ADVANCED PERFORMANCE IN WRITING 



Advanced writing at all levels will have an overall harmony in purpose and style that 
demonstrates a clarity of thought both in terms of the writer's central idea/focus and all the 
choices the writer makes to develop the text. Superior writing reflects evidence of sound 
decisions regarding organization and form, diction, tone, and the conventions of written 
English in relation to the intended meaning. 

These features of advanced writing hold true across grade levels; however, as children grow 
older, the breadth and depth of their experiences and development will be reflected in the 
quality of the decisions they make in their writing. 

Given the constraints of large scale assessment (such as a 25-minute time limits, surprise 
topic, lack of opportunity to explore and revise, and so forth), the following are indicators 
of advanced student performance. 

GRADE 4 

The advanced writing of fourth graders, for example, generally demonstrates predictable 
patterns of organization and a limited range of elaboration strategies. These papers are 
marked by experimentation with language, i.e., vocabulary, syntax, imagery, and so forth. 

GRADES 

Eighth grade writing, for example, shows a conscious experimentation both with organizing 
structures and with stylistic strategies (e.g., humor and irony). These papers demonstrate 
superior control of sentence structure and the conventions of language. Ideas are clearly 
stated. There is frequent experimentation with literary devices. 

GRADE 12 

Advanced writing on the twelfth grade level, for example, will demonstrate a mature tone, 
which possesses an obvious advantage of broader experience and perspective than might be 
seen among eighth grade samples. These writers will demonstrate a clearly reasoned, more 
objective sense of the world, themselves, and their experiences. An evident sophistication 
of style through fluency of expression, diction, syntax, tone, and so forth accompanies this 
increased sense of objective distance. 

Other evident traits are a clear mastery of the conventions of written English and a strong 
relational sense of structure and purpose. 
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Achievement Level Sample Papers 
Representing Student Writing Performance as Selected 
by the St. Louis Achievement Levels-Setting Panels (July 24-28, 1992) 
and/or by the Orlando Content Validity Study Group (September 4-6, 1992) 



Grade 4 
Story About a Magical Balloon 



Story About a Magic Balloon 01W8 
Books W6 1 , W62» W7 1 , W76 Grade 4 



Narrative/Imaginative 

SCORING GUIDE 
Primary Trait 

Primanr trait: Quality of narrative. 

Scoring rgtiQnglg: The direcuve asks respondents to write a story about an adventure with a magic balloon. Respondents 
may focus the narrative primarily on the balloon and its special qualities, or the balloon may play a more peripheral role. 

6 Extensively elaborated. In these responses, students present an engaging and well-written story about an 

adventure with a balloon. Their stories include inventive details about setting, characters, and events. Their stories 
have episodes that are well-developed, sequenced in clear and effective ways and resolved appropriately at the end. 
They contain more than one episode, where tension is developed and resolved several times. They usually have a 
clear beginning and ending, but when very elaborated, may be unfinished (because the student ran out of time). 

5 Elaborated. In these responses, students tell an engaging story about an imagined adventure with a balloon. 
Their stories include inventive details about setting, characters, and events. Their stories have episodes that are 
well-dcvelopcd, sequenced in clear and effective ways and resolved appropriately by the ending. These stories 
contain more than one episode developed in-depth through the establishment of tension in one episode and its 
resolution in another episode, or the presense of tention in two episodes but resolution in only one. They tend to 
have a clear beginning and ending. 

4 Developed. In these responses, students clearly show evidence of storytelling as they offer a narrative about an 
adventure with a balloon. Their stories include inventive details about setting, characters, and events. However, the 
story may begin or end abruptly. Overall, the events are minimally developed, but sequenced in a cohesive manner. 
Vie three main types are: (a) a story with a clear beginning and ending where one episode is developed in-depth 
through the use of tension: (b) a story with more than one episode (such as a series of things they would wish to do 
with the balloon) and a clear beginning and end: (c) a story with more than one episode is developed in-depth 
through the use of tension, but the beginning or end may be brief or missing. 

3 Minimally developed. In these responses, students address the imaginative situation in the task - having an 
adventure with a magical balloon or with a regular balloon. Their stories include a setting, characters, and a few 
events. In many of these stories only one thing happens or they visit several places but their narrative is primarily 
a dcscriptin of the places they have visited. These stories are not fully developed and their sequencing of events mav 
be unclear, disconnected, or contain gaps. Many simply list things they do with their balloon, but these lists are 
somewftat detailed and refer to a specific incident. 

2 Undeveloped response to task. In these responses, students address the imaginative situation in the task - 
havmg a magical balloon. They begin a story by describing a setung and/or opening event, but do not move beyond 
the bcgmnrng. They may describe somewhere they can go with the balloon, but nothing actually happens, or thev 
describe the things the balloon can do. but only refer to routine events. 

1 Response to topic. In these responses, students respond to the topic but do not appear to have understood the 
task. They may write about a balloon or about magic without describing an adventure with a special balloon. Or 
they rccopy text from the prompt, or tell a story that is unrelated to the prompt, hut is in narrative form. 



m BEST COPY AVAIUBLE 



In this section, you will have 25 minutes to write a story. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your story on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your story, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your story better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW, 



01W8 



Page 



^GIKOIKTinTHBNEXirPAGE^ 



GO 

17 



Section 



THIS PAGE LEFT BLANK ON PURPOSE. 



oiw« 



Gd|qi^nRTHBNEXi:PAGE^ 



Pretend that you have a magical balloon. This balloon helps vou have 
wonderful adventures. 

Write a story about one adventure you have with your magical balloon. 



UOCQHITI^THE KEXirPAG^ 



OIW8 



1» 



Page 3 



WP000024 



Section ^ 



A SPECIAL PAGE FOR IDEAS, NOTES, AND PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your story on the next page. 



0IW8 



Page 4 



fGCEOttTOcTHENEXTrPAGE^.^ 
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Section 



Begin your story on this page. 



PT 
PW 

ST 



Do not use. 
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Page 5 



;GO.,0MrTO.THENEXTt PAG^ 



21 



Section 



OlWB 



Page 6 
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OIW8 



Page 7 



23 



Grade 4 
Story About a Magical Balloon 

BASIC 



1^ I 



Begin your story on this page. 
VS/V^PV, X.\KjcX\\ \Vr, X^,-^ 'OAy ^(^y 

— r\,Cf^^ ^rXY^ — Ox^^ Vnau-C ' wn y ^.tx\ ^\V_"\0 Vr\^ 
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ST 



Do not use. 
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GQlOHimTHENEXrPAGEJ 



BEST COPY available" 



Grade 4 
Story About a Magical Balloon 

PROFICIENT 



Section 2, 



Begin your story on this page. 




L^Wn^.tii- c.^HLJi> JJiu^t^ "H .Hv, 
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PT 
PW 

ST 



Do not use. 
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OIWS 



Page 5 



GOrOltTO THE NEXT PAf 
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Grade 4 
Story About a Magical Balloon 

ADVANCED 
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Section 



Begin your story on this page. 
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PT 
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Do not use. 
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CttOrniTTHE NEXT PAGE. 
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Grade 4 
Spaceship 



Spaceship Oiwii 

Books W67, W68. W73. W77 Grade 4 



Persuasive 



SCORING GUIDE 
Primary Trait 



6 



primary tmif: Persuasion through articulation and support of a position. 

^ffilag^tmale: The directive requires respondents to take a position about whether the spaceship 
should be allowed to return to its planet, and to provide reasons to suppon their position. 

Extensively elaborated. In these responses, students aniculate a position for or against the 
return ot the spaceship, or suggest a plan, and they present an extended, well-written 
discussion on the reasons for their position. These responses may be similar to "5" responses 
but they are better organized, more clearly written, and less flawed. 

Elaborated. In these responses, students' aniculate a position for or against the return of the 
spaceship, or suggest a plan, and they provide an extended discussion on the reasons for 
ineir position. Or they present an argument based on a strong sense of audience. 

Developed. In these responses, students take a position for or against the return of the 
spaceship, or suggest a plan, and they discuss the reasons or give a brief argument for their 
position. Although the reasons rnay be more compelling or more clearly stated than in papers 
that received lower scores, the discussion may be unevenly developed. 

Minimally developed In these responses, students state or imply a position for o: against 
the return of the spaceship and give at least one substantive reason or several brief reasons to 
support theu- stand. Or, they present a plan for resolving the problem that includes some 
explanation about us benefits. These papers can be brief or have vague and confusing aspects. 
Note. Reasons can include humanitanan appeals (e.g., the scientists would not like to be 
studied, the aliens have a nght to live their own life.) 

Undeveloped response to task. In these responses, students state a position for or 
against the return of the spaceship or they present a plan for resolving the problem, but thev offer 
no reasons or only vague/inappropnate/irrelevant reasons to support their point of view Or 
they present confusing plans. Or, they present a reason or two on each side of the issue without 
articulating that they are undecided. wunoui 

Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task or they may appear to state a position but the paper is largely 
S?'rt';!.^h'- ^''^Pl^' ^hey may refer to spaceships in general or about aliens 
yisinng the earth without any indication of what to do about the problem or whv Or thev 
recopy text from the prompt. ^ ^' ' 
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Section 9 











In this section, you will have 25 minutes to write a letter. R^ad the 
assignment carefully and think about it before you begin Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your letter on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your letter, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your writing better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 



OlWI I 



Page 



[GIKOKTQiTHENEXTiPAGB; 
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Section 9 



A spaceship from another planet has landed on Earth. The creatures 
in the spaceship seem to be friendly and have let the space center in the 
United States know that their ship must return to their planet. Some 
scientists want to keep the spaceship on Earth and study the creatures 
Others thmk the creatures should be allowed to return to their own 



Write a letter to the director of the space center. Tell what vou think 
should be done with the creatures and their spaceship. Be sure to eive 
reasons to support your ideas. 



planet. 
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Section 2, 



A SPECIAL PAGE FOR IDEAS, NOTES, AND PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your letter on the next page. 
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Page 4 
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Begin your letter on this page. 
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Grade 4 
Spaceship 

BASIC 




Begin your letter on this page. 
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Begin your letter on this page. 
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Spaceship 
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Begin, your letter on tliis page. 
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Grade 4 
Favorite Story 
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Favorite Story 
Books W64, W65, W70, W74 

Informative/Analysis 

SCORING GUIDE 
Primary Trait 



01W5 

Grade 4 



Primary rmit: Summary of a story. 

Sgpringrntipn,^]?.: The directive asks respondents to tell about a favorite story. 



Extensively elaborated. In these responses, students identify a favorite story and provide a 
clear summary of this story. This summary is complete and contains relevant details about 
setting, characters, episodes, and ending. The features of the summary are well-organized, 
coherent, and unified. 

Elaborated. In these responses, students identify a favorite story and provide a clear summary 
of this story. This summary is complete; it mentions the major elements of the story: setting, 
characters, episodes, and an ending. It also contains relevant details about these elements. 

Developed. In these responses, students identify a favorite story and provide a clear summary 
of this story. This summary is complete; it mentions the major elements of the story: setting, 
characters, episodes, and an ending. 

Minimally developed. In these responses, students identify a favorite story and provide a 
partial summary of this story, but may not include all of the major story elements (setting, 
characters, episodes, and endings). 

Undeveloped response to task. In these responses, students identify a favorite story. 
Their summary is unclear, confused, and incomplete. 

Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task. They may refer to a particular story or stories without identifyins or 
describing a favorite story. Or, they recopy text from the prompt. 



0 = No response 

9 = Incomprehensible; totally off task: "I don't know. 




Section 2, 



In this section, you will have 25 minutes to write a paper. Rea J the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your paper on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your paper, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your paper better. 



PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 
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Think about a favorite story you have read or heard, or one you have 
seen in the movies or on television. Tell what the story is about for 
someone who does not know the story. Be sure to include interesting 
details about characters, places, events, or ideas. 
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A SPECIAL PAGE FOR IDEAS, NOTES, AJMD PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your paper on the next page. 
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Drug Searches at School 
Books W67. W68, W73, W77 
Books W67. W68, W73, W77 

Persuasive 

SCORING GUIDE 
Primary Trait 

Primary trait: Persuasion through suppon of position. 

Scoring rational: The directive asks students to write an essay in which they state their views on the 
proposed policy of random drug searches. 

6 Extensively elaborated. In these responses, students articulate their position for or against 
the proposed searches, or suggest an alternative plan, and they provide a cohrsive and developed 
argument explaining their position. These responses may be similar to "5" responses, but they 
are better organized, more clearly written, and less flawed. 

5 Elaborated. In these responses, students aniculate their position for or against the proposed 
searches, or suggest an alternative plan, and they provide an extended argument for their 
position. These are similar to "4" responses, but contain more information and elaboration. 

4 Developed. In these responses, students take a stand for or against the proposed searches, or 
suggest an alternative plan, and they provide reasons or an argument to suppon their position. 
They address both the issue of individual rights and the issue of controlling drugs either by 
addressing one and then the other or in a unified discussion. These responses may be unevenly 
developed (primarily concentrated on one issue or the other). 

3 Minimally developed. In these responses, students take a stand for or against the proposed 
searches, or suggest an alternative plan, and they provide a few reasons or a brief argument to 
suppon their position. These responses can center on the issue of individual rights, the issue of 
controlhng the drug problem, or provide some thoughts on both issues. 

2 Undeveloped response to task. In these responses, students take a stand for or against the 
proposed drug searches and provide at least one general or briefly stated reason to suppon the 
position, (often addressing one side or the other of the debate - - It would help controls 
drugs/It's an invasion of privacy.) Or, diey briefly review the existing plan agreeing/disagreeing 
or suggesting additions/modifications to various aspects. 

1 Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task. For example, they may only agree/disagree with die drug search plan 
or suggest an alternative plan without explaining why it is preferable. Or, they may suggest 
some small addition or modification to the drug searching procedures. They may discuss dru<^s 
or drug searches or their hostility about the drug searches, but they do not address the debate 
underiying the issue. Or, they recopy text from the prompt. 
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0 = No response 
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In this section, you will have 25 minutes to write an essay. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be iudged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your essay on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your essay, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your essay better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 



023Wn 



Page 1 



GQiOKiTtKTHe NEXT PAG^> 



Section ^ 



Because of concerns about dnig-related crime, your local school 
board^is considering a proposal that would allow administrators to 
search student lockers and personal belongings for drugs. To assist 
the administrators, drug-sniffing dogs would be used to help locate 
drugs in schools. Students found possessing drugs would be subject 
to arrest. 

What do you think about this proposal? Express your views in an 
essay that will be sent to the school board. Consider whether the 
proposal would affect individual rights and whether it would help 
control the potential drug problems in schools. 

Be sure to give reasons to support your ideas. 
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You may use this blank space 
to make notes and organize your ideas. 

Begin writing your essay on the next page. 
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Embarrassing Incident \ 023W6 

Books W60. W62, W69. W77 ^ Grade 8 

Books W60.W62,W69.W77 Grade 12 



Nanaiive;/Pasonal 

SCORING GUIDE 
Primary Trait 

Primary trait: Quality of narrative. 

Scoring mtifina Iff : llie directive asks respondents to tell about an embarrassing experience in such a way that 
someone who was not there would understand what made it so embarrassing. 

6 Extensively elaborated. In these responses, students provide a detailed, well-wriuen descripuon of ai 
cmbarrassmg experience. These responses may be simUar to "5" responses, but they are better organized 
more clearly wntten. and less flawed. Generally, these responses contain more developed. multiDle 
problems. 



Elaborated. In these responses, students provide a detailed story of an embarrassing experience providing 
an account of the experience and their feelings about what happened. These stories have more than one 
episode developed in depth through the use of tension (see tension under 4). Usually, this tension is 
developed in one episode and then resolved, or explained in a later episode. OccasionaUy. the writer may 
present tension in one episode, and another type of tension in a separate episode. Because these represent 
simple uses of tension, these papers stay at this level. 

Developed. In these responses, students tell a complete story about an embaiiassing experience in some 
detail. These stones have a clear beginning, middle, and ending, the events are listed in a sequential order 
where at least one episode is developed in detail through the use of tension {i.e. foreshadowing, extending 
the embarrassing incident itself, or by expanding on how the event made the writer feel, or discussing the 
result of the incident). This developement however, may be uneven or disjointed. 

Minimally developed. In these responses, students tell a brief story about an embarrassing experience 
but the story may be vague, rambling, or confusing. These stories do have a clear beginning, middle and ' 
ending, but none are elaborate or developed. 

Undeveloped response to task. In these responses, students tell about an embarrassing experience and 
provide little or no informauon about it. They may give few details, or give a report on the topic rather 
than tell a story. 

Response to topic. In these responses, students respond to the topic but do not appear to have 
understood the task. For example, they may refer to embarrassment without describing an incident Or 
• they recopy text from the prompt. Or they write a response that is unrelated to the prompt but is in ' 
narrative form. 



In this section, you will have 25 minutes to write a story. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your story on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your story, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your story better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 
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Section 3 



In our day-to-day lives, things sometimes happen that seem embarrassing at 
the time. But, in the telling, these stories can be funny. Write a story about 
an experience you have had that was embarrasting. Be sure to include enough 
details so that a reader could understand what made it embarrassing. 
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You may use this blank space 
to make notes and organize your ideas. 

Begin writing your story on the next page. 
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Books W64, W65, W70, W74 

Mbnnative/Report 



023W5 
Grades 
Grade 12 



SCORING GUIDE 
Primary Trait 



_ Primary trait: Description of an object and its uses. 



""^ ' ' , SgQring rari<?nnlc: The directive asks students to tell about an invented object and the need it is 



*^"ig"C^ to fulfill- Students may focus their descriptions in a variety of ways, emphasizing the need 
for the invention, its characteristics, or the advantages it offers. In scoring, readers should focus on 
i'C'^v** ^^'0' and organization of the information provided in the description 

6 Extensively elaborated. In these responses, students provide a cohesive, well-written 

description of an invention. This description is accompanied by an extensive discussion of the 
usefulness of, purpose of, or the perceived necessity for the invention. These responses may be 
similar to "5" responses, but they are better organized, more clearly written, and less flawed. 



Elaborated. In these responses, students provide a well-organized, detailed description of an 
invention, making its features and uses clear. 

Developed. In these responses, students describe an invention clearly, providing some 
information about its characteristics and identifying or implying the purposes or advantages of 
the invention. 

Minimally developed. In these responses, students name an invention and provide a 
description that is brief, general, or confusing in some (not all) respects. They may or may not 
allude to the advantages of the invention. 

Undeveloped response to task. In these responses, students identify an invention. They 
may attempt to describe it, but the information about the invention or its usefulness is either very 
abbreviated, irrelevant, or confusing. 

Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task. They may refer to inventions or address some remarks to the patent 
office. Or, they recopy text from the prompt. 



0 = No response 

9 = Incomprehensible; totally off task; "I don't know." 
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Section 3 



In this section; you will have 25 minutes to write a letter. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your letter on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your letter, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your writing better. 



PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 
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Have you ever been in the middle of some project or activity when 
you reaUzed that you needed or wanted something that does not exist? 

Think of something to invent. This can be almost anything, as long 
as it does not already exist, as far as you know. Or think of an existing 
object that needs to be significantly changed or improved — in other 
words, "reinvented." 

Write a letter to the United States Patent Office telling them about 
your invention. Be sure to explain how your invention will fulfill a 
particular need. 
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A SPECIAL PAGE FOR IDEAS, NOTES, AND PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your letter on the next page. 



023WS 



Page 4 



Gtt OmTff.THE N EXT PAGE^ 




Begin your letter on this page. 



PT 
PW 

ST 



Do not use. 

©OOOOOd)©®® 



023W5 



Page 5 



GaOMTOJHENECT:PAGB: 



23 



Section 3 



023W5 



FRir 



Page 6 

j ; 



GttOriTtrTHEHEXrPAGEi 



02JW5 



Page 7 



STOP 



2S 



Grade 8 
Invention 

BASIC 



z 



Section 2, ^ 



Begin your letter on this page. 



la 



, 



i I 



I 1 



Do not use. 

PT ©OCDCDOOOOCDCD 
PW ® o 

ST ©OCDCDOCDOOOCE) 



013W5 



Page 5 



GirOHTtTTHENEXTPABE: 



Grade 8 
Invention 



PROFICIENT 




^ Begin your letter on this page. 



'jtf?- on AO/mi A (i^yjj(ij.viAfim )^ Q/nA^miri. / 
c;(^S7Ub Jit). TiiAJmOh I'mndiL i 








Do not use. 

PT ®0©®0®®OCDCD 
PW ® O 

ST ©OOOOO®®®® 



oasws 



I ■••1. 



BEST COPY ftVAIL.ABl£ 



Pages 



Ga onraiHE next pacb^^ 



i S2 



Grade 8 
Invention 



ADVANCED 



Section ' 3 



rr 
r- 



If] 



ERIC 



A \ '0 ' 



Begin your letter on this page. 



{ « 
1 



I 



^ . (j ^ ^ .1 



j ! j s 



•^■'Jh 




i I A ' 



l.AJ>' CL.AJI. 0La-A-.|0 r%-A rW^ I ' 



J^lV^x/^^\l_ v'^Lkxyj-N v^A<vkQ n .-.j- 



■ I 




0X3W$ 









Do not use. 








FT 


CD 


O 


© © © ® 


© 


© 


© ® 


PW 


















CD 


® © © ® 


© 


© 


© © 


ST 




O 


© © © © 


© 


© 


© © 



Page 5 

BEST COPY MILStE 



Ca OKTILTHt NEXT PAGg^ 



Section. 



W/ u?Tx^^Ji J^yfcQL.tQj >^:ri^:'-i r?).'^-^^ /i_^'"o..><v.AA.i,>^i. ■ 




ERIC 



. 013W5 



Page 6 

1 O 



GICQKrmTHENEXIIPAGe^ 



'i 



Grade 12 
Embarrassing Incident 



Embarrassing Incident 023W6 

Books W60, W62, W69. W77 Grade 8 
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Nanative^exsonal 



SCORING GUIDE 
Primary Trait 



Primary trait! Quality of narrative. 



Scoring rationale : The directive asks respondents to tell about an embauassing experience in such a way that 
someone who was not there would understand what made it so embarrassing. 

6 Extensively elaborated. In these responses, students provide a detaUed, well-written description of an 
embarrassmg experience* These responses may be simUar to "5" responses, but they arc better organized, 
more clearly writien. and less Hawed, Generally, these responses contain more developed, multiple 
problems. 

5 Elaborated. In these responses, students provide a detailed story of an embarrassing experience, providing 
an account of the experience and their feelings about what happened. These stories have more than one 
episode developed in depth through the use of tension (see tension under 4). Usually, this tension is 
developed in one episode and then resolved, or explained in a later episode. Occasionally, the writer may 
present tension in one episode, and another type of tension in a separate episode. Because these represent 
simple uses of tension, these papers stay at this level. 

4 Developed. In these responses, students tell a complete story about an embarrassing experience in some 
detail. These stories have a clear beginning, middle, and ending, the events arc listed in a sequential order 
where at least one episode is developed in detail through the use of tension {U. foreshadowing, extending 
the embarrassing incident itself, or by expanding on how the event made the writer feel, or discussing the 
result of the incident). This developement however, may be uneven or disjointed. 

3 Minimally developed. In these responses, students tell a brief story about an embarrassing experience 
but the story may be vague, rambling, or confusing. These stories do have a clear beginning, middle, and ' 
ending, but none are elaborate or developed. 

2 Undeveloped response to task. In these responses, students tell about an embarrassing experience and 
provide little or no information about it. They may give few details, or give a report on the topic rather 
than tell a story. 

1 Response to topic. In these responses, students respond to the topic but do not appear to have 

understood the task. For example, they may refer to embarrassment without describing an incident. Or 
tiiey recopy text from the prompt. Or tiiey write a response that is unrelated to the prompt but is in 
narrative form. 



In this section, you will have 25 minutes to write a story. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your story on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your story, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your story better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 



023W6 



Page 1 



GO ON TO THE NEXT PACr 



ERIC 



'-8 



It 



Section ^ 



In our day-to-day lives, things sometimes happen that seem embarrassing at 
the time. But, in the telUng, these stories can be funny. Write a story about 
an experience you have had that was embarrassing. Be sure to include enough 
details so that a reader could understand what made it embarrassing. 



GaOMTOvTHEMEXTPAGE 



023W6 



^/ Page 3 



wpooooir 



iS9 



21 



Section 



A SPECIAL PAGE FOR IDEAS, NOTES, AND PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your story on the next page. 
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Invention 

Books W64, W65, W70, \V74 
Books W64, W65, W70, W74 

Informative/Report 

SCORING GUIDE 
Primary Trait 

Primary trait : Description of an object and its uses. 

Sgpring rationale: The directive asks students to tell about an invented object and the need it is 
designed to fulfill. Students may focus their descriptions in a variety of ways, emphasizing the need 
for the invention, its characteristics, or the advantages it offers. In scoring, readers should focus on 
the clarity and organization of the information provided in the description. 

6 Extensively elaborated. In these responses, students provide a cohesive, well-written 

description of an invention. This description is accompanied by an extensive discussion of the 
usefulness of, purpose of, or the perceived necessity for the invention. These responses may be 
similar to "5" responses, but they are better organized, more clearly written, and less flawed. 

5 Elaborated. In these responses, students provide a well-organized, detailed description of an 
invention, making its features and uses clear. 

4 Developed. In these responses, students describe an invention cleariy, providing some 

mformation about its characteristics and identifying or implying the purposes or advantages of 
the invention. 

3 Minimally developed. In these responses, students name an invention and provide a 

descnption that is brief, general, or confusing in some (not all) respects. They may or may not 
allude to the advantages of the invention. 

2 Undeveloped response to task. In these responses, students identify an invention. They 
may at; ^mpt to describe it, but the information about the invention or its usefulness is either very 
abbreviated, irrelevant, or confusing. 

1 Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task. They may refer to inventions or address some remarks to the patent 
office. Or, they recopy text from the prompt. 



023W5 

Grade 8 
Grade 12 



0 = No response 

9 = Incomprehensible; totally off task: "I don't know. 



SECTIONS 



Section 



In this section, you will have 25 minutes to write a letter. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your letter on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your letter, be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your writing better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 
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Section 



Have you ever been in the middle of some project or activity when 
you realized that you needed or wanted something that does not existt 

Think of something to invent. This can be almost anything, as long 
as it does not already exist, as far as you know. Or think of an existing 
object that needs to be significantly changed or improved — in other 
words, "reinvented." 

Write a letter to the United States Patent Office telling them about 
your invention. Be sure to explain how your invention will fulfill a 
particular need. 
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You may use this blank space 
to make notes and organize your ideas. 

Begin writing your letter on the next page. 
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Books W67, W68, W73, W77 
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Grade 8 
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Persuasive 



SCORING GUIDE 
Primary Trait 



Primary trait : Persuasion through support of position. 

Scoring rational: The directive asks students to write an essay in which they state their views on the 
proposed policy of random drug searches. 

6 Extensively elaborated. In these responses, students articulate their position for or against 
the proposed searches, or suggest an alternative plan, and they provide a cohesive and developed 
argument explaining their position. These responses may be similar to "5" responses, but they 
are better organized, more clearly written, and less flawed. 

5 Elaborated. In these responses, students articulate their position for or against the proposed 
searches, or suggest an alternative plan, and they provide an extended argument for their 
position. These are similar to "4" responses, but contain more information and elaboration. 

4 Developed. In these responses, students take a stand for or against the proposed searches, or 
suggest an alternative plan, and they provide reasons or an argument to suppon their position. 
They address both the issue of individual rights and the issue of controlling drugs either by 
addressing one and then the other or in a unified discussion. These responses may be unevenly 
developed (primarily concentrated on one issue or the other). 

3 Minimally developed. In these responses, students take a stand for or against the proposed 
searches, or suggest an alternative plan, and they provide a few reasons or a brief argument to 
support their position. These responses can center on the issue of individual rights, the issue of 
controlling the drug problem, or provide some thoughts on both issues. 

2 Undeveloped response to task. In these responses, students take a stand for or against the 
proposed drug searches and provide at least one general or briefly stated reason to support the 
position, (often addressing one side or the other of the debate - - It would help controls 
drugs/It's an invasion of privacy.) Or, they briefly review the existing plan agreeing/disagreeing 
or suggesting additions/modifications to various aspects. 

] Response to topic. In these responses, students respond to the topic but do not appear to 
have understood the task. For example, they may only agree/disagree with the drug search plan 
or suggest an alternative plan without explaining why it is preferable. Or, they may suggest 
some small addition or modification to the drug searching procedures. Thev may discuss drues 
or drug searches or their hostility about the drug searches, but they do not address the debate ^ 
underlying the issue. Or, they recopy text from the prompt. 



0 = No response 

9 = Incomprehensible; totally off task: "I don't know." 



In this section, you will have 25 minutes to write an essay. Read the 
assignment carefully and think about it before you begin. Be sure to 
respond to every part of the assignment. Your writing will be judged 
according to how well you develop your ideas. Remember that you can use 
the planning page to make notes and organize your ideas. 

Write your essay on the lined pages. Make your response as thoughtful 
and complete as possible. When you are writing your essay; be sure that 
your handwriting is clear. 

Do not go past the STOP sign at the end of the section. If you finish 
before time is called, you should go over your work again and change 
anything that you think will make your essay better. 

PLEASE TURN THE PAGE AND BEGIN THIS SECTION NOW. 
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Section 



Because of concerns about drug-related crime, your local school 
board IS conoidermg a proposal that would allow administrators to 
search student lockers and personal belongings for drugs. To assist 
the admmistrators, drug-sniffing dogs would be used to help locate 
drugs m schools. Students found possessing drugs would be subject 
to arrest. ' 

What do you think about this proposal? Express your views in an 
essay that wi 1 be sent to the school board. Consider whether the 
proposal would attect individual rights and whether it would help 
control the potential drug problems in schools. 

Be sure to give reasons to support your ideas. 
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A SPECIAL PAGE FOR IDEAS, NOTES, AND PLANS 

You may use this blank space 
to make notes and organize your ideas. 

Begin writing your essay on the next page. 
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